Multimodal Deep Learning for Activity and Context Recognition

Radu, Valentin; Tong, Catherine; Bhattacharya, Sourav; Lane, Nicholas D.; Mascolo, Cecilia; Marina, Mahesh K.; Kawsar, Fahim

doi:10.1145/3161174

Cited by 214 publications

(138 citation statements)

References 53 publications

Supporting

Mentioning

123

Contrasting

Unclassified

Order By: Relevance

“…Deep neural networks are composed of multiple (parameterized) non-linear transformations that are trained through a supervised or unsupervised objective function with the aim of yielding useful representations. ese techniques have achieved indisputable empirical success across a broad spectrum of problems [4,21,28,35,43,56,65,66] thanks to the increasing dataset sizes and computing power availability. Nevertheless, representation learning still stands as a fundamental problem in machine intelligence and is an active area of research (see [8] for a detailed survey).…”

Section: Representation Learningmentioning

confidence: 99%

See 1 more Smart Citation

Multi-task Self-Supervised Learning for Human Activity Detection

Saeed

Özçelebi

Lukkien

2019

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

229

199

View full text Add to dashboard Cite

Deep learning methods are successfully used in applications pertaining to ubiquitous computing, pervasive intelligence, health, and well-being. Speci cally, the area of human activity recognition (HAR) is primarily transformed by the convolutional and recurrent neural networks, thanks to their ability to learn semantic representations directly from raw input. However, in order to extract generalizable features massive amounts of well-curated data are required, which is a notoriously challenging task; hindered by privacy issues and annotation costs. erefore, unsupervised representation learning (i.e., learning without manually labeling the instances) is of prime importance to leverage the vast amount of unlabeled data produced by smart devices. In this work, we propose a novel self-supervised technique for feature learning from sensory data that does not require access to any form of semantic labels, i.e., activity classes. We learn a multi-task temporal convolutional network to recognize transformations applied on an input signal. By exploiting these transformations, we demonstrate that simple auxiliary tasks of the binary classi cation result in a strong supervisory signal for extracting useful features for the down-stream task. We extensively evaluate the proposed approach on several publicly available datasets for smartphone-based HAR in unsupervised, semi-supervised and transfer learning se ings. Our method achieves performance levels superior to or comparable with fully-supervised networks trained directly with activity labels, and it performs signi cantly be er than unsupervised learning through autoencoders. Notably, for the semi-supervised case, the self-supervised features substantially boost the detection rate by a aining a kappa score between 0.7 − 0.8 with only 10 labeled examples per class. We get similar impressive performance even if the features are transferred from a di erent data source. Self-supervision drastically reduces the requirement of labeled activity data, e ectively narrowing the gap between supervised and unsupervised techniques for learning meaningful representations. While this paper focuses on HAR as the application domain, the proposed approach is general and could be applied to a wide variety of problems in other areas. 000:2 • A. Saeed et al.activity recognition (HAR), 1D convolutional and recurrent neural networks trained on raw labeled signals signi cantly improve the detection rate over traditional methods [20,44,68,72,73]. Despite the recent advances in the eld of HAR, learning representations from a massive amount of unlabeled data still presents a signi cant challenge. Obtaining large, well-curated activity recognition datasets is problematic due to a number of issues. First, smartphone data are privacy sensitive, which makes it hard to collect su cient amounts of user-activity instances in a real-life se ing. Second, the annotation cost and the time it takes to generate a large volume of labeled instances are prohibitive. Finally, the diversity of devices, types of embedd...

show abstract

Section: Representation Learningmentioning

confidence: 99%

“…Over the last years, deep neural networks have been widely adopted for time-series and sensory data processing; achieving impressive performance in several application areas pertaining to pervasive sensing, ubiquitous computing, industries, health and well-being [17,21,38,56,60,73]. In particular, for smartphone-based human Fig.…”

Section: Introductionmentioning

confidence: 99%

Multi-task Self-Supervised Learning for Human Activity Detection

Saeed

Özçelebi

Lukkien

2019

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

229

199

View full text Add to dashboard Cite

show abstract

“…Their goals are similar, while their learning processes are different. Both can be exploited to extract patterns from unlabeled mobile data, which may be subsequently employed for various supervised learning tasks, e.g., routing [186], mobile activity recognition [187], [188], periocular verification [189] and base station user number prediction [190].…”

Section: Auto-encodersmentioning

confidence: 99%

Deep Learning in Mobile and Wireless Networking: A Survey

Zhang

Patras

Haddadi

2019

IEEE Commun. Surv. Tutorials

1,335

838

View full text Add to dashboard Cite

The rapid uptake of mobile devices and the rising popularity of mobile applications and services pose unprecedented demands on mobile and wireless networking infrastructure. Upcoming 5G systems are evolving to support exploding mobile traffic volumes, real-time extraction of fine-grained analytics, and agile management of network resources, so as to maximize user experience. Fulfilling these tasks is challenging, as mobile environments are increasingly complex, heterogeneous, and evolving. One potential solution is to resort to advanced machine learning techniques, in order to help manage the rise in data volumes and algorithm-driven applications. The recent success of deep learning underpins new and powerful tools that tackle problems in this space.In this paper we bridge the gap between deep learning and mobile and wireless networking research, by presenting a comprehensive survey of the crossovers between the two areas. We first briefly introduce essential background and state-of-theart in deep learning techniques with potential applications to networking. We then discuss several techniques and platforms that facilitate the efficient deployment of deep learning onto mobile systems. Subsequently, we provide an encyclopedic review of mobile and wireless networking research based on deep learning, which we categorize by different domains. Drawing from our experience, we discuss how to tailor deep learning to mobile environments. We complete this survey by pinpointing current challenges and open future directions for research.

show abstract

“…MagTrack also detects aggressive steering motions that can indicate fatigue driving. Simultaneous Tracking and Classification: STC was initially proposed to solve the target tracking and classification in the radar monitoring scenario [26,46,47,53]. The basic idea is to use the special feature metrics to infer the target type.…”

Section: State Of the Artmentioning

confidence: 99%

MagTrack

Huang

Chen

Lin

2019

Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services

View full text Add to dashboard Cite

Hands on the wheel, eyes on the road" is the central guideline of safe vehicle driving practices. Many advanced driver assistance systems can effectively detect abnormal vehicle motions. However, these systems often leave insufficient time for drivers to respond to complex road situations, especially when the drivers are distracted. To reduce accidents, it is essential to detect whether a driver complies with safe driving guidelines in real time and provide warnings early before any dangerous maneuvers occur. There are vision-based driver distraction monitoring systems which rely on cameras in high-end vehicles, but their performances are heavily constrained by visibility requirements. In this paper, we present MagTrack, a driver monitoring system that is based on tracking magnetic tags worn by the user. With a single smartwatch and two low-cost magnetic accessories: a hand magnetic ring and a head magnetic eyeglasses clip, our system tracks and classifies a driver's bimanual and head movements simultaneously using both analytical and approximation sensing models. Our approach is robust to driver's postures, vehicles, and environmental changes. We demonstrate that a wide range of activities can be detected by our system, including bimanual steering, visual and manual distractions, and lane changes and turns. In extensive road tests with 500+ instances of driving activities and 500+ minutes of road driving with 10 subjects, MagTrack achieves 87% of precision and 90% of recall rate on the detection of unsafe driving activities. CCS CONCEPTS• Human-centered computing → Ubiquitous and mobile computing.

show abstract

Multimodal Deep Learning for Activity and Context Recognition

Cited by 214 publications

References 53 publications

Multi-task Self-Supervised Learning for Human Activity Detection

Multi-task Self-Supervised Learning for Human Activity Detection

Deep Learning in Mobile and Wireless Networking: A Survey

MagTrack

Contact Info

Product

Resources

About