Imaging and fusing time series for wearable sensor-based human activity recognition

Qin, Zhen; Zhang, Hongjie; Meng, Shuyu; Choo, Kim‐Kwang Raymond

doi:10.1016/j.inffus.2019.06.014

Cited by 195 publications

(92 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human activity recognition has been developed mostly using computer vision techniques over optical sensors data, 1,68 however there is a growing body of research based on wearable sensors. 69,70 Data fusion for human activity recognition often refers to same kind of sensors, e.g. the fusion of IMUs tracking data in Ref.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Improved Activity Recognition Combining Inertial Motion Sensors and Electroencephalogram Signals

Graña

Aguilar-Moreno

Asiaín

et al. 2020

Int. J. Neur. Syst.

View full text Add to dashboard Cite

Human activity recognition and neural activity analysis are the basis for human computational neureoethology research dealing with the simultaneous analysis of behavioral ethogram descriptions and neural activity measurements. Wireless electroencephalography (EEG) and wireless inertial measurement units (IMU) allow the realization of experimental data recording with improved ecological validity where the subjects can be carrying out natural activities while data recording is minimally invasive. Specifically, we aim to show that EEG and IMU data fusion allows improved human activity recognition in a natural setting. We have defined an experimental protocol composed of natural sitting, standing and walking activities, and we have recruited subjects in two sites: in-house ([Formula: see text]) and out-house ([Formula: see text]) populations with different demographics. Experimental protocol data capture was carried out with validated commercial systems. Classifier model training and validation were carried out with scikit-learn open source machine learning python package. EEG features consist of the amplitude of the standard EEG frequency bands. Inertial features were the instantaneous position of the body tracked points after a moving average smoothing to remove noise. We carry out three validation processes: a 10-fold cross-validation process per experimental protocol repetition, (b) the inference of the ethograms, and (c) the transfer learning from each experimental protocol repetition to the remaining repetitions. The in-house accuracy results were lower and much more variable than the out-house sessions results. In general, random forest was the best performing classifier model. Best cross-validation results, ethogram accuracy, and transfer learning were achieved from the fusion of EEG and IMUs data. Transfer learning behaved poorly compared to classification on the same protocol repetition, but it has accuracy still greater than 0.75 on average for the out-house data sessions. Transfer leaning accuracy among repetitions of the same subject was above 0.88 on average. Ethogram prediction accuracy was above 0.96 on average. Therefore, we conclude that wireless EEG and IMUs allow for the definition of natural experimental designs with high ecological validity toward human computational neuroethology research. The fusion of both EEG and IMUs signals improves activity and ethogram recognition.

show abstract

Section: Discussionmentioning

confidence: 99%

“…the fusion of IMUs tracking data in Ref. 70. To our knowledge, there has not been any previous attempt to apply the fusion of EEG and other sensor data.…”

Section: Discussionmentioning

confidence: 99%

Improved Activity Recognition Combining Inertial Motion Sensors and Electroencephalogram Signals

Graña

Aguilar-Moreno

Asiaín

et al. 2020

Int. J. Neur. Syst.

View full text Add to dashboard Cite

show abstract

“…Other authors demonstrated that human actions can be better described using motion and speed. Murad and Ryun 2017 [15] and Qin et al [16] applied body-worn sensors and Long Short Term Memory (LSTM) Recurrent Neural Networks for human motion description. The latter involves gyroscope and accelerometer measures.…”

Section: Related Workmentioning

confidence: 99%

Prediction of Human Activities Based on a New Structure of Skeleton Features and Deep Learning Model

Jaouedi¹,

Perales

Buades

et al. 2020

Sensors

View full text Add to dashboard Cite

The recognition of human activities is usually considered to be a simple procedure. Problems occur in complex scenes involving high speeds. Activity prediction using Artificial Intelligence (AI) by numerical analysis has attracted the attention of several researchers. Human activities are an important challenge in various fields. There are many great applications in this area, including smart homes, assistive robotics, human–computer interactions, and improvements in protection in several areas such as security, transport, education, and medicine through the control of falling or aiding in medication consumption for elderly people. The advanced enhancement and success of deep learning techniques in various computer vision applications encourage the use of these methods in video processing. The human presentation is an important challenge in the analysis of human behavior through activity. A person in a video sequence can be described by their motion, skeleton, and/or spatial characteristics. In this paper, we present a novel approach to human activity recognition from videos using the Recurrent Neural Network (RNN) for activity classification and the Convolutional Neural Network (CNN) with a new structure of the human skeleton to carry out feature presentation. The aims of this work are to improve the human presentation through the collection of different features and the exploitation of the new RNN structure for activities. The performance of the proposed approach is evaluated by the RGB-D sensor dataset CAD-60. The experimental results show the performance of the proposed approach through the average error rate obtained (4.5%).

show abstract

“…Time series classification is a thriving area of study. Existing algorithms find applications in computer‐aided decision‐making systems, online monitoring in areas such as human activity recognition , automation and control , remote sensing , manufacturing , astronomy , and many other areas of science and engineering.…”

Section: Literature Reviewmentioning

confidence: 99%

Lagged encoding for image‐based time series classification using convolutional neural networks

Jastrzębska

2020

Statistical Analysis

View full text Add to dashboard Cite

Time series classification is a thriving area of research in machine learning. Among many applications, it is frequently applied to human activity analysis. Time series describing a human in motion are ubiquitously collected via omnipresent mobile devices and can be subjected to further processing. In this paper, we propose a novel, deep learning approach to time series classification. It is based on a lagged time series representation stored as images andConvolutional Neural Network used to image classification. We present a comparative study on different variants of lagged time series representation and we evaluate their effectiveness in a series of empirical experiments. We show that the developed method provides satisfying classification accuracy. The proposed image-based time series encoding is less resource-consuming than encodings used in other image-based approaches to time series classification. It is worth to emphasize that the proposed time series encoding conceals original time series values. Images are saved without scales and the order of observations cannot be reconstructed. Thus, the method is particularly suitable for systems that need to store sensitive information. K E Y W O R D S classification, convolutional neural network, image recognition, time series 1 Stat Anal Data Min: The ASA Data Sci Journal. 2020;13:245-260. wileyonlinelibrary.com/sam

show abstract

Imaging and fusing time series for wearable sensor-based human activity recognition

Cited by 195 publications

References 16 publications

Improved Activity Recognition Combining Inertial Motion Sensors and Electroencephalogram Signals

Improved Activity Recognition Combining Inertial Motion Sensors and Electroencephalogram Signals

Prediction of Human Activities Based on a New Structure of Skeleton Features and Deep Learning Model

Lagged encoding for image‐based time series classification using convolutional neural networks

Contact Info

Product

Resources

About