On-Device Deep Personalization for Robust Activity Data Collection

Abstract

One of the biggest challenges of activity data collection is the need to rely on users and keep them engaged to continually provide labels. Recent breakthroughs in mobile platforms have proven effective in bringing deep neural networks powered intelligence into mobile devices. This study proposes a novel on-device personalization for data labeling for an activity recognition system using mobile sensing. The key idea behind this system is that estimated activities personalized for a specific individual user can be used as feedback to motivate user contribution and improve data labeling quality. First, we exploited fine-tuning using a Deep Recurrent Neural Network to address the lack of sufficient training data and minimize the need for training deep learning on mobile devices from scratch. Second, we utilized a model pruning technique to reduce the computation cost of on-device personalization without affecting the accuracy. Finally, we built a robust activity data labeling system by integrating the two techniques outlined above, allowing the mobile application to create a personalized experience for the user. To demonstrate the proposed model’s capability and feasibility, we developed and deployed the proposed system to realistic settings. For our experimental setup, we gathered more than 16,800 activity windows from 12 activity classes using smartphone sensors. We empirically evaluated the proposed quality by comparing it with a baseline using machine learning. Our results indicate that the proposed system effectively improved activity accuracy recognition for individual users and reduced cost and latency for inference for mobile devices. Based on our findings, we highlight critical and promising future research directions regarding the design of efficient activity data collection with on-device personalization.

Effects of sliding window variation in the performance of acceleration-based human activity recognition using deep learning models

“…As for the inference time, in Mairittha, Mairittha & Inoue (2021) ) it is calculated in two of the models presented here: LSTM and CNN-LSTM. The inference times obtained were 0.0106s and 0.3941s, respectively, resulting in LSTM’s is around 3X slower than CNN-LSTM’s.…”

“…It is a more common approach in conventional ML methods, but some authors define a window size segmentation as pre-processing of input for HAR with DL methods. Mairittha, Mairittha & Inoue (2021) performed data labeling for an activity recognition system using inertial (acceleration and angular velocity) mobile sensing in both simple-LSTM and hybrid CNN-LSTM using 5.12s window size (at 20 Hz). With overlapping, this window size represents around 100 frames.…”

Machine Learning for Healthcare Wearable Devices: The Big Picture

“…Additionally, the computational complexity for inference and for online training on the device for personalization is a concern due to the limited computation power for wearable devices until now. On-device deep learning and transfer learning for personalization have been researched in [ 43 , 109 – 111 ].…”

A systematic review of smartphone-based human activity recognition methods for health research

“…The identified differences are later quantified through various statistical measures to distinguish between activities. In an alternative approach, the process of feature extraction is automated using deep learning, which handles feature selection using simple signal processing units, called neurons, that have been arranged in a network structure that is multiple layers deep 59,[68][69][70] . As with many applications of deep learning, the results may not be easily interpretable.…”

“…Incremental learning techniques were proposed to adapt the classification model to new data streams and unseen activities [103][104][105] . Other semi-supervised approaches were proposed to utilize unlabeled data to improve the personalization of HAR systems 106 and data annotation 53,70 . To increase the effectiveness of HAR, some studies used a hierarchical approach, where the classification was performed in separate stages and each stage could use a different classifier.…”