Sensor Fusion Approach for Multiple Human Motion Detection for Indoor Surveillance Use-Case

Abstract

Multi-human detection and tracking in indoor surveillance is a challenging task due to various factors such as occlusions, illumination changes, and complex human-human and human-object interactions. In this study, we address these challenges by exploring the benefits of a low-level sensor fusion approach that combines grayscale and neuromorphic vision sensor (NVS) data. We first generate a custom dataset using an NVS camera in an indoor environment. We then conduct a comprehensive study by experimenting with different image features and deep learning networks, followed by a multi-input fusion strategy to optimize our experiments with respect to overfitting. Our primary goal is to determine the best input feature types for multi-human motion detection using statistical analysis. We find that there is a significant difference between the input features of optimized backbones, with the best strategy depending on the amount of available data. Specifically, under a low-data regime, event-based frames seem to be the preferred input feature type, while higher data availability benefits the combined use of grayscale and optical flow features. Our results demonstrate the potential of sensor fusion and deep learning techniques for multi-human tracking in indoor surveillance, although it is acknowledged that further studies are needed to confirm our findings.

Fusion Object Detection and Action Recognition to Predict Violent Action

“…There are several developments regarding to the monitoring of human motion using sensor fusion, such as the work presented by Abbasi et al [ 13 ] although it is does not focuses for the in-car monitoring, demonstrating the potential of using sensor fusion and deep learning techniques for multi-human tracking by using NVS sensors. In [ 14 ], Melo et al use a similar approach of sensor fusion, in this case, by combining RGB and thermal cameras with deep learning techniques for monitoring the presence of masks on people in public spaces in the context of COVID-19 and for measuring the body temperature.…”