Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

Abstract

Human activity recognition (HAR) is becoming increasingly important, especially with the growing number of elderly people living at home. However, most sensors, such as cameras, do not perform well in low-light environments. To address this issue, we designed a HAR system that combines a camera and a millimeter wave radar, taking advantage of each sensor and a fusion algorithm to distinguish between confusing human activities and to improve accuracy in low-light settings. To extract the spatial and temporal features contained in the multisensor fusion data, we designed an improved CNN-LSTM model. In addition, three data fusion algorithms were studied and investigated. Compared to camera data in low-light environments, the fusion data significantly improved the HAR accuracy by at least 26.68%, 19.87%, and 21.92% under the data level fusion algorithm, feature level fusion algorithm, and decision level fusion algorithm, respectively. Moreover, the data level fusion algorithm also resulted in a reduction of the best misclassification rate to 2%~6%. These findings suggest that the proposed system has the potential to enhance the accuracy of HAR in low-light environments and to decrease human activity misclassification rates.

Thermal image enhancement by artificial multiscale-exposure image fusion

“…They provide a wealth of information about the temperature of the observed object, which is more resilient to variations in illumination or background clutter compared to traditional cameras. Thermal imaging cameras have been successfully utilized in various computer vision tasks, including pedestrian detection [1], behavior recognition [2], military surveillance [3], autonomous navigation in GPS-denied environments [4], fire detection [5], heat leakage detection [6], gas leak detection [7], and more. However, thermal sensors capable of capturing high-resolution images tend to be expensive.…”

Detection of Sensors Used for Adversarial Examples against Machine Learning Models

“…For instance, Namazi et al developed a method to detect and track continuous objects surrounding vehicles, locate objects in front of the vehicle, and mitigate occlusion issues to provide more precise readings [4]. Similarly, multiple systems, Zhou et al [5] and Yuan et al [6] proposed systems to recognize human activities in low-light indoor conditions by using multiple sensor devices.…”