Counting Activities Using Weakly Labeled Raw Acceleration Data: A Variable-Length Sequence Approach with Deep Learning to Maintain Event Duration Flexibility

Abstract

This paper presents a novel approach for counting hand-performed activities using deep learning and inertial measurement units (IMUs). The particular challenge in this task is finding the correct window size for capturing activities with different durations. Traditionally, fixed window sizes have been used, which occasionally result in incorrectly represented activities. To address this limitation, we propose segmenting the time series data into variable-length sequences using ragged tensors to store and process the data. Additionally, our approach utilizes weakly labeled data to simplify the annotation process and reduce the time to prepare annotated data for machine learning algorithms. Thus, the model receives only partial information about the performed activity. Therefore, we propose an LSTM-based architecture, which takes into account both the ragged tensors and the weak labels. To the best of our knowledge, no prior studies attempted counting utilizing variable-size IMU acceleration data with relatively low computational requirements using the number of completed repetitions of hand-performed activities as a label. Hence, we present the data segmentation method we employed and the model architecture that we implemented to show the effectiveness of our approach. Our results are evaluated using the Skoda public dataset for Human activity recognition (HAR) and demonstrate a repetition error of ±1 even in the most challenging cases. The findings of this study have applications and can be beneficial for various fields, including healthcare, sports and fitness, human–computer interaction, robotics, and the manufacturing industry.

A Novel Framework Based on Deep Learning Architecture for Continuous Human Activity Recognition with Inertial Sensors

“…The input layer is fed by a multidimensional array whose shape is (N w , W l , N c ), where N w is the number of windows in the input dataset, which may change across subjects and trials, W l is the window length, which is fixed, and N c is the number of sensor channels, which differs according to the combination of sensors to be evaluated (e.g., it is six for any sensor pair, nine for any sensor triple, and twelve for the combination with four sensors). A grid-search method is employed to optimize the architectural characteristic of the 1D-CNN, i.e., determine the number of convolutional and dense layers, as well as the number of neurons that maximize validation accuracy [9,38]. Therefore, each branch consists of two 1D convolutional layers using 128 filters and kernels of size 5 for the first and 3 for the second one, and one max-pooling layer.…”

“…The scientific literature has given more and more attention to the field of human activity recognition (HAR), which aims to classify human actions by exploiting sensor data [7]. HAR has covered various contexts, from industry [8,9] to sport [10], but a wider application lies in the medical field [7,[10][11][12][13][14]: in this realm, subjects' activities can be remotely registered outside the clinic [15] and clinicians can evaluate their functional abilities after treatment [16,17]. HAR can also enhance a rehabilitative program inside the clinic for the sake of an assist-as-needed approach: in particular, recognizing the motor actions performed by patients (e.g., post-stroke individuals or people with psychomotor dysfunction) can allow for correcting motions or encouraging further exercise when required [18].…”

“…A typical HAR experimental protocol encompasses a set of activities that the subject is asked to perform. These motor tasks may involve mainly upper body [9,20], or lower body (e.g., walking or climbing/descending stairs) [15,17,21,22], or even require the individual to drive upper-and lower-extremities in a proper combination (e.g., lying in bed) [23][24][25][26][27][28][29]. Furthermore, the protocol to collect data for HAR purposes tends to be designed such that ADLs are performed separately, i.e., a batch of repetitions of one of the activities to be recognized is asked to be executed by every subject [30].…”

“…The recognition of human actions in a workflow targeting HAR requires a pipeline entailing a sequence of steps that may include data processing, feature extraction and artificial intelligence (AI) techniques to perform classification: at first, the signals acquired from sensors are processed to reduce noise [41], cope with missing values and remove possible artifacts [9,42]; secondly, data are segmented to identify the portion of the preprocessed signals that are informative of the executed activities [43]; signals can be optionally converted into images as well [15,17,20,28,44,45]; afterward, features are extracted for each segment from either images or time-series data [1,2,38,46] to capture meaningful characteristics of the performed activities; ultimately, these features and their corresponding ground truth labels are used as input to train a classifier, whose performance is evaluated based on quantitative criteria, such as accuracy [47].…”

Robust Feature Representation Using Multi-Task Learning for Human Activity Recognition

“…HAR is an active field of research in pervasive computing that aims to detect human physical activities through machine learning models. HAR has various applications in healthcare [ 1 , 2 ], sports [ 3 , 4 ], industry [ 5 , 6 , 7 ], and other fields. Commonly, HAR models utilize activity signals recorded by wearable or visual sensors.…”