Multimodal ambulatory sleep detection

Chen, Weixuan; Sano, Akane; Martinez, Daniel; Taylor, Sara; McHill, Andrew W.; Phillips, Andrew J. K.; Klerman, Elizabeth B.; Picard, Rosalind W.

doi:10.1109/bhi.2017.7897306

Cited by 14 publications

(10 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Applications have been seen in audio-visual speech recognition [52], image captioning [63], machine translation [34], sentiment analysis [55] and affect recognition [30]. In the space of ubiquitous computing, example applications include human activity recognition [1], sleep detection [12] and emotion recognition [36]. Many recognition tasks were previously only primarily performed with unimodal learning, with the availability of low-energy sensors, many such tasks are recently explored using multimodal learning.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Deep Learning for Activity and Context Recognition

Radu

Tong

Bhattacharya

et al. 2018

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

215

123

View full text Add to dashboard Cite

Wearables and mobile devices see the world through the lens of half a dozen low-power sensors, such as, barometers, accelerometers, microphones and proximity detectors. But differences between sensors ranging from sampling rates, discrete and continuous data or even the data type itself make principled approaches to integrating these streams challenging. How, for example, is barometric pressure best combined with an audio sample to infer if a user is in a car, plane or bike? Critically for applications, how successfully sensor devices are able to maximize the information contained across these multi-modal sensor streams often dictates the fidelity at which they can track user behaviors and context changes. This paper studies the benefits of adopting deep learning algorithms for interpreting user activity and context as captured by multi-sensor systems. Specifically, we focus on four variations of deep neural networks that are based either on fully-connected Deep Neural Networks (DNNs) or Convolutional Neural Networks (CNNs). Two of these architectures follow conventional deep models by performing feature representation learning from a concatenation of sensor types. This classic approach is contrasted with a promising deep model variant characterized by modality-specific partitions of the architecture to maximize intra-modality learning. Our exploration represents the first time these architectures have been evaluated for multimodal deep learning under wearable data -- and for convolutional layers within this architecture, it represents a novel architecture entirely. Experiments show these generic multimodal neural network models compete well with a rich variety of conventional hand-designed shallow methods (including feature extraction and classifier construction) and task-specific modeling pipelines, across a wide-range of sensor types and inference tasks (four different datasets). Although the training and inference overhead of these multimodal deep approaches is in some cases appreciable, we also demonstrate the feasibility of on-device mobile and wearable execution is not a barrier to adoption. This study is carefully constructed to focus on multimodal aspects of wearable data modeling for deep learning by providing a wide range of empirical observations, which we expect to have considerable value in the community. We summarize our observations into a series of practitioner rules-of-thumb and lessons learned that can guide the usage of multimodal deep learning for activity and context detection.

show abstract

Section: Related Workmentioning

confidence: 99%

Multimodal Deep Learning for Activity and Context Recognition

Radu

Tong

Bhattacharya

et al. 2018

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

215

123

View full text Add to dashboard Cite

show abstract

“…We use the long short-term memory recurrent neural nets (LSTM-RNNs) [18], as they have recently shown great success in sequence learning tasks such as speech recognition [12] and sleep/wake classification [6]. In this work, we employ a bidirectional LSTM-RNNs architecture [6] (shown in Fig.1) to estimate the PSPI values from input features X (i.e., a sequence of facial landmarks). While the traditional RNNs are unable to learn temporal dependencies longer than a few time steps due to the vanishing gradient problem [17], LSTM-RNNs overcome this by introducing recurrently connected memory blocks instead of traditional neural network nodes.…”

Section: Rnns For Pspi Estimationmentioning

confidence: 99%

Personalized Automatic Estimation of Self-Reported Pain Intensity from Facial Expressions

Martinez

Rudovic

Picard

2017

2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Self Cite

View full text Add to dashboard Cite

Pain is a personal, subjective experience that is commonly evaluated through visual analog scales (VAS). While this is often convenient and useful, automatic pain detection systems can reduce pain score acquisition efforts in large-scale studies by estimating it directly from the participants' facial expressions. In this paper, we propose a novel two-stage learning approach for VAS estimation: first, our algorithm employs Recurrent Neural Networks (RNNs) to automatically estimate Prkachin and Solomon Pain Intensity (PSPI) levels from face images. The estimated scores are then fed into the personalized Hidden Conditional Random Fields (HCRFs), used to estimate the VAS, provided by each person. Personalization of the model is performed using a newly introduced facial expressiveness score, unique for each person. To the best of our knowledge, this is the first approach to automatically estimate VAS from face images. We show the benefits of the proposed personalized over traditional non-personalized approach on a benchmark dataset for pain analysis from face images.

show abstract

“…In this paper, we present a novel automated method to detect sleep/wake state and sleep onset/offset times using recurrent neural networks with long short-term memory (LSTM) cells [30] applied to multimodal data from a smartphone and a wrist-worn sensor, and labeled by both actigraphy and sleep diaries. The proposed method combines multimodal ambulatory physiological and behavioral data and improves upon our team's earlier work showing that refining sleep onset/offset times can help improve sleep characterization [31]. This paper makes several novel contributions:…”

Section: Introductionmentioning

confidence: 88%

“…In our previous paper [31], we proposed a sleep episode onset/offset detector using crosscorrelation-based template matching. Specifically, we first computed the sleep probability patterns (for both sleep onset and offset) from the sleep detection results within the training set.…”

Section: F Sleep Episode Onset/offset Detection: Template-matching-based Methods Vs Differential Bidirectional Lstm Model (Participant-dementioning

confidence: 99%

Multimodal Ambulatory Sleep Detection Using LSTM Recurrent Neural Networks

Sano

Chen

Lopez-Martinez

et al. 2019

IEEE J. Biomed. Health Inform.

Self Cite

View full text Add to dashboard Cite

Unobtrusive and accurate ambulatory methods are needed to monitor long-term sleep patterns for improving health. Previously developed ambulatory sleep detection methods rely either in whole or in part on self-reported diary data as ground truth, which is a problem since people often do not fill them out accurately. This paper presents an algorithm that uses multimodal data from smartphones and wearable technologies to detect sleep/wake state and sleep onset/offset using a type of recurrent neural network with long-short-term memory (LSTM) cells for synthesizing temporal information. We collected 5580 days of multimodal data from 186 participants and compared the new method for sleep/wake classification and sleep onset/offset detection to (1) non-temporal machine learning methods and (2) a state-of-the-art actigraphy software. The new LSTM method achieved a sleep/wake classification accuracy of 96.5%, and sleep onset/offset detection F1 scores of 0.86 and 0.84 respectively, with mean absolute errors of 5.0 and 5.5 min, respectively, when compared with sleep/wake state and sleep onset/offset assessed using actigraphy and sleep diaries. The LSTM results were statistically superior to those from non-temporal machine learning algorithms and the actigraphy software. We show good generalization of the new algorithm by comparing participant-dependent and participant-independent models, and we show how to make the model nearly realtime with slightly reduced performance.

show abstract

Multimodal ambulatory sleep detection

Cited by 14 publications

References 13 publications

Multimodal Deep Learning for Activity and Context Recognition

Multimodal Deep Learning for Activity and Context Recognition

Personalized Automatic Estimation of Self-Reported Pain Intensity from Facial Expressions

Multimodal Ambulatory Sleep Detection Using LSTM Recurrent Neural Networks

Contact Info

Product

Resources

About