Classifying Imbalanced Multi-modal Sensor Data for Human Activity Recognition in a Smart Home using Deep Learning

Abstract

In smart homes, data generated from real-time sensors for human activity recognition is complex, noisy and imbalanced. It is a significant challenge to create machine learning models that can classify activities which are not as commonly occurring as other activities. Machine learning models designed to classify imbalanced data are biased towards learning the more commonly occurring classes. Such learning bias occurs naturally, since the models better learn classes which contain more records. This paper examines whether fusing real-world imbalanced multi-modal sensor data improves classification results as opposed to using unimodal data; and compares deep learning approaches to dealing with imbalanced multi-modal sensor data when using various resampling methods and deep learning models. Experiments were carried out using a large multi-modal sensor dataset generated from the Sensor Platform for HEalthcare in a Residential Environment (SPHERE). The data comprises 16104 samples, where each sample comprises 5608 features and belongs to one of 20 activities (classes). Experimental results using SPHERE demonstrate the challenges of dealing with imbalanced multi-modal data and highlight the importance of having a suitable number of samples within each class for sufficiently training and testing deep learning models. Furthermore, the results revealed that when fusing the data and using the Synthetic Minority Oversampling Technique (SMOTE) to correct class imbalance, CNN-LSTM achieved the highest classification accuracy of 93.67% followed by CNN, 93.55%, and LSTM, i.e. 92.98%.

AUTO-HAR: An adaptive human activity recognition framework using an automated CNN architecture design

“…Researchers have been experimenting with DL based methods to develop tools for detecting normal or abnormal human movements using sensor-collected data or images [41] . The literature varies in its categorization of human actions based on whether they are produced by handwork (conventional-based) [29] or algorithmic recognition (artificial intelligence) using Deep neural networks (DNNs) [18] or other techniques [30] . DNNs and their main types have received a vast amount of effort for computer vision problems.…”

A Comparison of Sampling Methods for Dealing with Imbalanced Wearable Sensor Data in Human Activity Recognition using Deep Learning

“…In the field of Human Activity Recognition (HAR), addressing imbalanced data presents a significant challenge, a common issue observed in various public datasets, including Opportunity [8], WISDM V1.1 [9], SPHERE [10] and PAMAP2 [11]. Imbalanced data can profoundly affect the performance of deep learning models utilized in HAR tasks.…”

Deep multi-view learning methods: A review

“…Human actions in videos or images usually consist of highly articulated motions, human-object interactions and complicated temporal structures [178,179,180,181,182]. Song et al [178] present a new action recognition framework under multi-modal scenarios based on deep CNN and RNN architectures and can better learn effective feature representations for action classification.…”

“…Classifying imbalanced multi-modal sensor data in the environment of smart home for activity recognition is quite challenging. To this end, Alani et al [179] first examine the effectiveness of using multi-modal data and then compare deep learning methods with other methods in addressing the imbalanced multi-modal data. Trumble et al [180] propose a deep CNN based human performance capture system for the challenging marker-less pose estimation from multi-view videos.…”