WiWeHAR: Multimodal Human Activity Recognition Using Wi-Fi and Wearable Sensing Modalities

Muaaz, Muhammad; Chelli, Ali; Abdelgawwad, Ahmed; Mallofré, Andreu Català; Pätzold, Matthias

doi:10.1109/access.2020.3022287

Cited by 44 publications

(42 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The raw CSI data contains amplitude and phase information, which are corrupted by noise; and therefore, the raw CSI data streams cannot effectively be used to extract micro-Doppler signatures [24]. The CSI data amplitude is mainly corrupted by the ambient noise and adaptive changes of the transmission parameters [41].…”

Section: Processing Of Channel State Information and Spectrogram Computationmentioning

confidence: 99%

Section: Phase Correctionmentioning

confidence: 99%

“…Inspired by the previous studies [24,25,42], we use the CSI ratio method [42] in this work. The CSI ratio method is more economical and simpler to set up, because it does not require additional hardware compared to the back-toback channel configuration method [24]. Moreover, our experiments and previous works [24,25] suggest that the CSI ratio method is more effective in reducing the phase distortions and noise from the amplitude information, compared to the phase sanitization and back-to-back channel configuration methods.…”

Section: Phase Correctionmentioning

confidence: 99%

“…Fig. 3 The falling activity spectrograms obtained a without and b with applying the CSI ratio-based phase calibration technique where H i,j (f k , t) is denotes the time-variant channel transfer function (CTF) 2 of a transmission link between ith transmit and j th receive antenna pair sampled at the kth subcarrier f k [24]. The used CSI Tool provides estimated CSIdata for K = 30 OFDM subcarriers.…”

Section: Phase Correctionmentioning

confidence: 99%

See 3 more Smart Citations

Wi-Sense: a passive human activity recognition system using Wi-Fi and convolutional neural network and its integration in health information systems

et al. 2021

Self Cite

View full text Add to dashboard Cite

A human activity recognition (HAR) system acts as the backbone of many human-centric applications, such as active assisted living and in-home monitoring for elderly and physically impaired people. Although existing Wi-Fi-based human activity recognition methods report good results, their performance is affected by the changes in the ambient environment. In this work, we present Wi-Sense—a human activity recognition system that uses a convolutional neural network (CNN) to recognize human activities based on the environment-independent fingerprints extracted from the Wi-Fi channel state information (CSI). First, Wi-Sense captures the CSI by using a standard Wi-Fi network interface card. Wi-Sense applies the CSI ratio method to reduce the noise and the impact of the phase offset. In addition, it applies the principal component analysis to remove redundant information. This step not only reduces the data dimension but also removes the environmental impact. Thereafter, we compute the processed data spectrogram which reveals environment-independent time-variant micro-Doppler fingerprints of the performed activity. We use these spectrogram images to train a CNN. We evaluate our approach by using a human activity data set collected from nine volunteers in an indoor environment. Our results show that Wi-Sense can recognize these activities with an overall accuracy of 97.78%. To stress on the applicability of the proposed Wi-Sense system, we provide an overview of the standards involved in the health information systems and systematically describe how Wi-Sense HAR system can be integrated into the eHealth infrastructure.

show abstract

Section: Processing Of Channel State Information and Spectrogram Computationmentioning

confidence: 99%

Section: Phase Correctionmentioning

confidence: 99%

Section: Phase Correctionmentioning

confidence: 99%

Section: Phase Correctionmentioning

confidence: 99%

See 2 more Smart Citations

Wi-Sense: a passive human activity recognition system using Wi-Fi and convolutional neural network and its integration in health information systems

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…There are many ways of integrating modalities for activity recognition task [44][45][46][47][48][49][50][51][52][53][54][55][56][57], with the 3 major groups being sensor-level [45,49,50], feature-level [44,57], and decision-level [51][52][53][54][55][56] fusion. Fusion at the decision level is the most frequently used method which takes advantage of training machine learning and deep learning models for each modality.…”

Section: Limitations In Existing Literaturementioning

confidence: 99%

Deep Learning–Based Multimodal Data Fusion: Case Study in Food Intake Episodes Detection Using Wearable Sensors

Bahador¹,

Ferreira²,

Tamminen³

et al. 2021

JMIR Mhealth Uhealth

View full text Add to dashboard Cite

Background Multimodal wearable technologies have brought forward wide possibilities in human activity recognition, and more specifically personalized monitoring of eating habits. The emerging challenge now is the selection of most discriminative information from high-dimensional data collected from multiple sources. The available fusion algorithms with their complex structure are poorly adopted to the computationally constrained environment which requires integrating information directly at the source. As a result, more simple low-level fusion methods are needed. Objective In the absence of a data combining process, the cost of directly applying high-dimensional raw data to a deep classifier would be computationally expensive with regard to the response time, energy consumption, and memory requirement. Taking this into account, we aimed to develop a data fusion technique in a computationally efficient way to achieve a more comprehensive insight of human activity dynamics in a lower dimension. The major objective was considering statistical dependency of multisensory data and exploring intermodality correlation patterns for different activities. Methods In this technique, the information in time (regardless of the number of sources) is transformed into a 2D space that facilitates classification of eating episodes from others. This is based on a hypothesis that data captured by various sensors are statistically associated with each other and the covariance matrix of all these signals has a unique distribution correlated with each activity which can be encoded on a contour representation. These representations are then used as input of a deep model to learn specific patterns associated with specific activity. Results In order to show the generalizability of the proposed fusion algorithm, 2 different scenarios were taken into account. These scenarios were different in terms of temporal segment size, type of activity, wearable device, subjects, and deep learning architecture. The first scenario used a data set in which a single participant performed a limited number of activities while wearing the Empatica E4 wristband. In the second scenario, a data set related to the activities of daily living was used where 10 different participants wore inertial measurement units while performing a more complex set of activities. The precision metric obtained from leave-one-subject-out cross-validation for the second scenario reached 0.803. The impact of missing data on performance degradation was also evaluated. Conclusions To conclude, the proposed fusion technique provides the possibility of embedding joint variability information over different modalities in just a single 2D representation which results in obtaining a more global view of different aspects of daily human activities at hand, and yet preserving the desired performance level in activity recognition.

show abstract

Proposed spatio‐temporal features for human activity classification using ensemble classification model

Tyagi

Singh

Dev

2023

Concurrency and Computation

View full text Add to dashboard Cite

Classifying human actions from still images or video sequences is a demanding task owing to issues, like lighting, backdrop clutter, variations in scale, partial occlusion, viewpoint, and appearance. A lot of appliances, together with video systems, human-computer interfaces, and surveillance necessitate a compound action recognition system. Here, the proposed system develops a novel scheme for HAR. Initially, filtering as well as background subtraction is done during preprocessing. Then, the features including local binary pattern (LBP), bag of the virtual word (BOW), and the proposed local spatio-temporal features are extracted. Then, in the recognition phase, an ensemble classification model is introduced that includes Recurrent Neural networks (RNN 1 and RNN 2) and Multi-Layer Perceptron (MLP 1 and MLP 2). The features are classified using RNN 1 and RNN 2, and the outputs from RNN 1 and RNN 2 are further classified using MLP 1 and MLP 2, respectively. Finally, the outputs attained from MLP 1 and MLP 2 are averaged and the final classified output is obtained. At last, the superiority of the developed approach is proved on varied measures. K E Y W O R D Saccuracy, human action, multi-layer perceptron, proposed spatio-temporal features, RNN INTRODUCTION"HAR is defined as a classification task where the movement of a person is categorized based on data received from different sources such as sensors, cameras, etc.". It has different appliances in healthcare, particularly in constant action monitoring and fall recognition of aged persons. [1][2][3] In addition, it has vast prospective to support emerging appliances namely, augmented reality, IoT, and indoor localization in smart construction controlling systems to offer a secure indoor surrounding with higher EE.Diverse studies were carried out on HAR techniques in the state of the works using computer vision, ambient devices, sensors, and smartphones. These approaches are usually classified into two classes, that is, passive and active monitoring schemes. [4][5][6] In an AMS, exterior devices like wearable sensors and cameras were deployed. Sensor-based HAR is extensively deployed in analysis, in which a person has to continually carry sensors such as gyroscope, pedometer, and accelerometer that can turn out to be difficult for the person in numerous cases, particularly for elder ones. [7][8][9]

show abstract

WiWeHAR: Multimodal Human Activity Recognition Using Wi-Fi and Wearable Sensing Modalities

Cited by 44 publications

References 51 publications

Wi-Sense: a passive human activity recognition system using Wi-Fi and convolutional neural network and its integration in health information systems

Wi-Sense: a passive human activity recognition system using Wi-Fi and convolutional neural network and its integration in health information systems

Deep Learning–Based Multimodal Data Fusion: Case Study in Food Intake Episodes Detection Using Wearable Sensors

Proposed spatio‐temporal features for human activity classification using ensemble classification model

Contact Info

Product

Resources

About