Robust Unsupervised Factory Activity Recognition with Body-worn Accelerometer Using Temporal Structure of Multiple Sensor Data Motifs

Xia, Qingxin; Korpela, Joseph; Namioka, Yasuo; Maekawa, Taira

doi:10.1145/3411836

Cited by 27 publications

(4 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Xia et al . [ 57 ] proposed complex activity recognition models by utilizing sensor data motifs , which discretize sensor signals into symbols that can be fed into sequencing models. However, these works on complex activity recognition require accurate labeling of sensory time series data, which is tedious.…”

Section: Related Workmentioning

confidence: 99%

X-Char

Jeyakumar

Sarker

García

et al. 2023

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

End-to-end deep learning models are increasingly applied to safety-critical human activity recognition (HAR) applications, e.g., healthcare monitoring and smart home control, to reduce developer burden and increase the performance and robustness of prediction models. However, integrating HAR models in safety-critical applications requires trust, and recent approaches have aimed to balance the performance of deep learning models with explainable decision-making for complex activity recognition. Prior works have exploited the compositionality of complex HAR (i.e., higher-level activities composed of lower-level activities) to form models with symbolic interfaces, such as concept-bottleneck architectures, that facilitate inherently interpretable models. However, feature engineering for symbolic concepts-as well as the relationship between the concepts-requires precise annotation of lower-level activities by domain experts, usually with fixed time windows, all of which induce a heavy and error-prone workload on the domain expert. In this paper, we introduce X-CHAR, an eXplainable Complex Human Activity Recognition model that doesn't require precise annotation of low-level activities, offers explanations in the form of human-understandable, high-level concepts, while maintaining the robust performance of end-to-end deep learning models for time series data. X-CHAR learns to model complex activity recognition in the form of a sequence of concepts. For each classification, X-CHAR outputs a sequence of concepts and a counterfactual example as the explanation. We show that the sequence information of the concepts can be modeled using Connectionist Temporal Classification (CTC) loss without having accurate start and end times of low-level annotations in the training dataset-significantly reducing developer burden. We evaluate our model on several complex activity datasets and demonstrate that our model offers explanations without compromising the prediction accuracy in comparison to baseline models. Finally, we conducted a mechanical Turk study to show that the explanations provided by our model are more understandable than the explanations from existing methods for complex activity recognition.

show abstract

Section: Related Workmentioning

confidence: 99%

X-Char

Jeyakumar

Sarker

García

et al. 2023

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

show abstract

“…Maekawa et al [21] measured the duration of each operation period of a factory worker who wears an acceleration sensor by tracking a motif that appears only once in each operation period using a particle filter. Xia et al [22], [23] estimated the starting and ending times of each operation in an operation period based on the tracked motif using information derived from process instructions. In contrast, our motif discovery approach is based on calculating the similarity between sensor data segments with location labels to discover motifs specific to a location class based on the concept of the Gini impurity.…”

Section: Motif Detectionmentioning

confidence: 99%

IndoLabel: Predicting Indoor Location Class by Discovering Location-Specific Sensor Data Motifs

et al. 2022

Self Cite

View full text Add to dashboard Cite

This study presents a method for predicting location classes of a room such as a kitchen, and restroom, where a user is located by discovering locationspecific sensor data motifs in sensor data observed by user's sensor devices, such as smartwatch, without requiring labeled training data collected in a target environment. For example, we can observe similar waveforms corresponding to kitchen knife chopping actions using body-worn accelerometers in kitchens and can also observe similar sound features by active sound probing in bathrooms because of their water-resistant walls. This indicates that such location-specific sensor data motifs can be inherent information for location class prediction in almost every environment. This study proposes a novel method that automatically detects location-specific motifs from time series sensor data by calculating a score that represents the "location specificity" of each motif in a time series. Previous studies on location class prediction assume that locationspecific sensor data are always observed in a room or use handcrafted rules and templates to detect location-specific sensor data resulting in difficulties in applying them to several realistic environments. In contrast, our method, named IndoLabel, can automatically discover short sensor data motifs, specific to a location class, and can automatically build an environment-independent location classifier without requiring handcrafted rules and templates. The proposed method was evaluated in real house environments using leave-one-environment-out cross-validation and achieved a state-of-theart performance although labeled training data in the target environment was unavailable.

show abstract

“…Thanks to this approach, the data analyzed was transformed into an appropriate, internal representation (feature vector) from which the learning subsystem can detect or classify patterns [35]. The DL techniques based on Artificial Neural Networks (ANNs) are used in human activity recognition systems [36], systems with Myo bands [33,37], or with sensors such as Microsoft Kinect [38,39] and Intel RealSense [40], accelerometer, gyroscope, or sensors of mobile devices (such as smartphone) [41] or wearable devices [42] as well as in systems based on video analysis [11]. To recognize a worker in action, smart manufacturing systems may use the ultrasonic sensors, Inertial Measurement Unit (IMU) [43], and the surface electromyography (sEMG) signals obtained from a Myo armband with Discrete Fourier Transformation (DFT) and Convolutional Neural Network (CNN) [37].…”

Section: Research Literaturementioning

confidence: 99%

An Automated Recognition of Work Activity in Industrial Manufacturing Using Convolutional Neural Networks

2021

View full text Add to dashboard Cite

The automated assessment and analysis of employee activity in a manufacturing enterprise, operating in accordance with the concept of Industry 4.0, is essential for a quick and precise diagnosis of work quality, especially in the process of training a new employee. In the case of industrial solutions, many approaches involving the recognition and detection of work activity are based on Convolutional Neural Networks (CNNs). Despite the wide use of CNNs, it is difficult to find solutions supporting the automated checking of work activities performed by trained employees. We propose a novel framework for the automatic generation of workplace instructions and real-time recognition of worker activities. The proposed method integrates CNN, CNN Support Vector Machine (SVM), CNN Region-Based CNN (Yolov3 Tiny) for recognizing and checking the completed work tasks. First, video recordings of the work process are analyzed and reference video frames corresponding to work activity stages are determined. Next, work-related features and objects are determined using CNN with SVM (achieving 94% accuracy) and Yolov3 Tiny network based on the characteristics of the reference frames. Additionally, matching matrix between the reference frames and the test frames using mean absolute error (MAE) as a measure of errors between paired observations was built. Finally, the practical usefulness of the proposed approach by applying the method for supporting the automatic training of new employees and checking the correctness of their work done on solid fuel boiler equipment in a manufacturing company was demonstrated. The developed information system can be integrated with other Industry 4.0 technologies introduced within an enterprise.

show abstract

Robust Unsupervised Factory Activity Recognition with Body-worn Accelerometer Using Temporal Structure of Multiple Sensor Data Motifs

Cited by 27 publications

References 36 publications

X-Char

X-Char

IndoLabel: Predicting Indoor Location Class by Discovering Location-Specific Sensor Data Motifs

An Automated Recognition of Work Activity in Industrial Manufacturing Using Convolutional Neural Networks

Contact Info

Product

Resources

About