Classical and deep learning methods for recognizing human activities and modes of transportation with smartphone sensors

Gjoreski, Martin; Janko, Vito; Slapničar, Gašper; Mlakar, Miha; Reščič, Nina; Bizjak, Jani; Drobnič, Vid; Marinko, Matej; Mlakar, Nejc; Luštrek, Mitja; Gams, Matjaž

doi:10.1016/j.inffus.2020.04.004

Cited by 79 publications

(74 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The DNN architecture used in this study is deep Spectrotemporal ResNet. A similar Spectro-temporal ResNet architecture has already proved successful for human activity recognition in our previous study [14] by achieving comparable accuracy to state-of-the-art feature-based models. The structure is based on an idea for training very deep end-to-end networks for image recognition; i.e., it uses shortcut (residual) connections to fight the gradient-vanishing problem [44].…”

Section: B Deep Learningmentioning

confidence: 91%

“…The classic ML approach learns from a large body of expert-defined features, and the DL approach learns both from a time-domain (the raw PCG signal) representation of the signal and a temporaldomain representation (the spectrogram) of the signal. This approach was successful in our previous study of human activity recognition from smartphone sensor data [14].…”

Section: Introductionmentioning

confidence: 87%

“…We present the analysis of the method's performance with respect to the segment (window) size. (ii) The proposed end-to-end DL architecture learns both from the temporal representation of the signal and the spectral representation of the signal, whereas most of the approaches in the related works use end-to-end learning in one of the domains only (either spectral or temporal [14]). (iii) We used the PhysioNet Challenge datasets to evaluate our approach and to provide a comparison with the challenge baseline method; additionally, we used our own dataset, which, in addition to including the typical healthy vs. patient labels, is also labeled for the specific CHF phase, i.e., compensated (when the patient feels well) and decompensated (when the patient does not feel well) for some of the patients.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Machine Learning and End-to-End Deep Learning for the Detection of Chronic Heart Failure From Heart Sounds

et al. 2020

Self Cite

View full text Add to dashboard Cite

Chronic heart failure (CHF) affects over 26 million of people worldwide, and its incidence is increasing by 2% annually. Despite the significant burden that CHF poses and despite the ubiquity of sensors in our lives, methods for automatically detecting CHF are surprisingly scarce, even in the research community. We present a method for CHF detection based on heart sounds. The method combines classic Machine-Learning (ML) and end-to-end Deep Learning (DL). The classic ML learns from expert features, and the DL learns from a spectro-temporal representation of the signal. The method was evaluated on recordings from 947 subjects from six publicly available datasets and one CHF dataset that was collected for this study. Using the same evaluation method as a recent PhysoNet challenge, the proposed method achieved a score of 89.3, which is 9.1 higher than the challenge's baseline method. The method's aggregated accuracy is 92.9% (error of 7.1%); while the experimental results are not directly comparable, this error rate is relatively close to the percentage of recordings labeled as ''unknown'' by experts (9.7%). Finally, we identified 15 expert features that are useful for building ML models to differentiate between CHF phases (i.e., in the decompensated phase during hospitalization and in the recompensated phase) with an accuracy of 93.2%. The proposed method shows promising results both for the distinction of recordings between healthy subjects and patients and for the detection of different CHF phases. This may lead to the easier identification of new CHF patients and the development of home-based CHF monitors for avoiding hospitalizations. INDEX TERMS Chronic heart failure, deep learning, heart sounds, machine learning, PCG.

show abstract

Section: B Deep Learningmentioning

confidence: 91%

Section: Introductionmentioning

confidence: 87%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Machine Learning and End-to-End Deep Learning for the Detection of Chronic Heart Failure From Heart Sounds

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Another idea that can be explored is the training of person-specific classification models. Moreover, we intend to attempt to further increase the classification performance of the method by combining it with a hidden Markov model, or similar, which would also take into account information on the temporal dependencies of sequential windows [ 58 ]. Lastly, we plan to incorporate energy-optimization techniques for conserving the battery power of the sensors, which is one of the main constraints on smartwatches and similar wrist-worn devices [ 59 ].…”

Section: Discussionmentioning

confidence: 99%

Detection of Gait Abnormalities for Fall Risk Assessment Using Wrist-Worn Inertial Sensors and Deep Learning

Kiprijanovska

Gjoreski

Gams

2020

Sensors

Self Cite

View full text Add to dashboard Cite

Falls are a significant threat to the health and independence of elderly people and represent an enormous burden on the healthcare system. Successfully predicting falls could be of great help, yet this requires a timely and accurate fall risk assessment. Gait abnormalities are one of the best predictive signs of underlying locomotion conditions and precursors of falls. The advent of wearable sensors and wrist-worn devices provides new opportunities for continuous and unobtrusive monitoring of gait during daily activities, including the identification of unexpected changes in gait. To this end, we present in this paper a novel method for determining gait abnormalities based on a wrist-worn device and a deep neural network. It integrates convolutional and bidirectional long short-term memory layers for successful learning of spatiotemporal features from multiple sensor signals. The proposed method was evaluated using data from 18 subjects, who recorded their normal gait and simulated abnormal gait while wearing impairment glasses. The data consist of inertial measurement unit (IMU) sensor signals obtained from smartwatches that the subjects wore on both wrists. Numerous experiments showed that the proposed method provides better results than the compared methods, achieving 88.9% accuracy, 90.6% sensitivity, and 86.2% specificity in the detection of abnormal walking patterns using data from an accelerometer, gyroscope, and rotation vector sensor. These results indicate that reliable fall risk assessment is possible based on the detection of walking abnormalities with the use of wearable sensors on a wrist.

show abstract

“…The spectro-temporal ResNet (STRNet) is a special case of the mid-fusion approach. in previous study on human activity recognition from smartphone sensors [71], for chronic heart failure detection from heart sounds [72], and for blood pressure estimation from photoplethysmogram (PPG) data [73].…”

Section: ) Deep Learing Architecturesmentioning

confidence: 99%

Machine Learning and End-to-End Deep Learning for Monitoring Driver Distractions From Physiological and Visual Signals

et al. 2020

Self Cite

View full text Add to dashboard Cite

It is only a matter of time until autonomous vehicles become ubiquitous; however, human driving supervision will remain a necessity for decades. To assess the driver's ability to take control over the vehicle in critical scenarios, driver distractions can be monitored using wearable sensors or sensors that are embedded in the vehicle, such as video cameras. The types of driving distractions that can be sensed with various sensors is an open research question that this study attempts to answer. This study compared data from physiological sensors (palm electrodermal activity (pEDA), heart rate and breathing rate) and visual sensors (eye tracking, pupil diameter, nasal EDA (nEDA), emotional activation and facial action units (AUs)) for the detection of four types of distractions. The dataset was collected in a previous driving simulation study. The statistical tests showed that the most informative feature/modality for detecting driver distraction depends on the type of distraction, with emotional activation and AUs being the most promising. The experimental comparison of seven classical machine learning (ML) and seven end-to-end deep learning (DL) methods, which were evaluated on a separate test set of 10 subjects, showed that when classifying windows into distracted or not distracted, the highest F1-score of 79% was realized by the extreme gradient boosting (XGB) classifier using 60-second windows of AUs as input. When classifying complete driving sessions, XGB's F1-score was 94%. The best-performing DL model was a spectro-temporal ResNet, which realized an F1-score of 75% when classifying segments and an F1-score of 87% when classifying complete driving sessions. Finally, this study identified and discussed problems, such as label jitter, scenario overfitting and unsatisfactory generalization performance, that may adversely affect related ML approaches.

show abstract

Classical and deep learning methods for recognizing human activities and modes of transportation with smartphone sensors

Cited by 79 publications

References 22 publications

Machine Learning and End-to-End Deep Learning for the Detection of Chronic Heart Failure From Heart Sounds

Machine Learning and End-to-End Deep Learning for the Detection of Chronic Heart Failure From Heart Sounds

Detection of Gait Abnormalities for Fall Risk Assessment Using Wrist-Worn Inertial Sensors and Deep Learning

Machine Learning and End-to-End Deep Learning for Monitoring Driver Distractions From Physiological and Visual Signals

Contact Info

Product

Resources

About