Video2IMU: Realistic IMU features and signals from videos

Abstract

Human Activity Recognition (HAR) from wearable sensor data identifies movements or activities in unconstrained environments. HAR is a challenging problem as it presents great variability across subjects. Obtaining large amounts of labelled data is not straightforward, since wearable sensor signals are not easy to label upon simple human inspection. In our work, we propose the use of neural networks for the generation of realistic signals and features using human activity monocular videos. We show how these generated features and signals can be utilized, instead of their real counterparts, to train HAR models that can recognize activities using signals obtained with wearable sensors. To prove the validity of our methods, we perform experiments on an activity recognition dataset created for the improvement of industrial work safety. We show that our model is able to realistically generate virtual sensor signals and features usable to train a HAR classifier with comparable performance as the one trained using real sensor data. Our results enable the use of available, labeled video data for training HAR models to classify signals from wearable sensors.

Co-simulation of human digital twins and wearable inertial sensors to analyse gait event estimation

“…A similar approach was chosen by Lämsä et al (2022), who used neural networks with VIDEO2IMU to generate IMU signals and features from monocular videos of human activities. Their results suggested that HAR systems trained using virtual sensor data could perform considerably better than baseline models, trained using only physical IMU data (Kwon et al, 2021;Lämsä et al, 2022). Esteban et al (2017) deployed GANs to synthesise respiratory data, where a generator model was used to augment data, and a discriminator attempts to distinguish between real and artificial data.…”

“… Kwon et al (2020 , 2021) presented IMUTube, an automated processing pipeline for human activity recognition (HAR) that integrates existing computer vision and signal processing techniques to convert video of human activity into virtual streams of IMU data. A similar approach was chosen by Lämsä et al (2022) , who used neural networks with VIDEO2IMU to generate IMU signals and features from monocular videos of human activities. Their results suggested that HAR systems trained using virtual sensor data could perform considerably better than baseline models, trained using only physical IMU data ( Kwon et al, 2021 ; Lämsä et al, 2022 ).…”

Combining 3-D Human Pose Estimation and IMU Sensors for Human Identification and Tracking in Multiperson Environments

“…However, none of these studies investigated on the fusion of HPE and IMUs data for identification and tracking of the 3D human skeleton associated with the individual wearing the IMUs in real and multi-person scenarios. Other works combine video and IMU to generate inertial data from human poses, with the aim of mitigating the lack of labeled training data [9], [10].…”