Inertial Sensor Data to Image Encoding for Human Action Recognition

Hossain

2022

Neural Comput & Applic

Capturing time and frequency relationships of time series signals offers an inherent barrier for automatic human activity recognition (HAR) from wearable sensor data. Extracting spatiotemporal context from the feature space of the sensor reading sequence is challenging for the current recurrent, convolutional, or hybrid activity recognition models. The overall classification accuracy also gets affected by large size feature maps that these models generate. To this end, in this work, we have put forth a hybrid architecture for wearable sensor data-based HAR. We initially use Continuous Wavelet Transform to encode the time series of sensor data as multi-channel images. Then, we utilize a Spatial Attention-aided Convolutional Neural Network (CNN) to extract higher-dimensional features. To find the most essential features for recognizing human activities, we develop a novel feature selection (FS) method. In order to identify the fitness of the features for the FS, we first employ three filter-based methods: Mutual Information (MI), Relief-F, and minimum redundancy maximum relevance (mRMR). The best set of features is then chosen by removing the lower-ranked features using a modified version of the Genetic Algorithm (GA). The K-Nearest Neighbors (KNN) classifier is then used to categorize human activities. We conduct comprehensive experiments on five well-known, publicly accessible HAR datasets, namely UCI-HAR, WISDM, MHEALTH, PAMAP2, and HHAR. Our model significantly outperforms the state-of-the-art models in terms of classification performance. We also observe an improvement in overall recognition accuracy with the use of GA-based FS technique with a lower number of features. The source code of the paper is publicly available here.

Section: Related Workmentioning

confidence: 55%

Human activity recognition from sensor data using spatial attention-aided CNN with genetic algorithm

Hossain

2022

Neural Comput & Applic

“…Even though depth datasets contain one-channel information, they provide object structure and background information with two different colors, making them confusing and time-consuming to process for lightweight devices. Inertial datasets [ 15 ] provide more compact information for human action. They contain acceleration and gyroscope information along the x-, y-, and z-axes, which can be processed easily.…”

Section: Introductionmentioning

confidence: 99%

Dynamic Edge Convolutional Neural Network for Skeleton-Based Human Action Recognition

Tasnim

Baek

2023

Sensors

To provide accessible, intelligent, and efficient remote access such as the internet of things, rehabilitation, autonomous driving, virtual games, and healthcare, human action recognition (HAR) has gained much attention among computer vision researchers. Several methods have already been addressed to ensure effective and efficient action recognition based on different perspectives including data modalities, feature design, network configuration, and application domains. In this article, we design a new deep learning model by integrating criss-cross attention and edge convolution to extract discriminative features from the skeleton sequence for action recognition. The attention mechanism is applied in spatial and temporal directions to pursue the intra- and inter-frame relationships. Then, several edge convolutional layers are conducted to explore the geometric relationships among the neighboring joints in the human body. The proposed model is dynamically updated after each layer by recomputing the graph on the basis of k-nearest joints for learning local and global information in action sequences. We used publicly available benchmark skeleton datasets such as UTD-MHAD (University of Texas at Dallas multimodal human action dataset) and MSR-Action3D (Microsoft action 3D) to evaluate the proposed method. We also investigated the proposed method with different configurations of network architectures to assure effectiveness and robustness. The proposed method achieved average accuracies of 99.53% and 95.64% on the UTD-MHAD and MSR-Action3D datasets, respectively, outperforming state-of-the-art methods.

“…To address these issues, some studies have demonstrated that fusing multiple images (i.e., multimodal fusion) can achieve higher performance than using a single image [39,40]. However, how to fuse multimodal features effectively is still challenging.…”

Section: Introductionmentioning

confidence: 99%

Meta-Feature Fusion for Few-Shot Time Series Classification

2023

Deep learning has been widely adopted for end-to-end time-series classification (TSC). However, the effectiveness of deep learning heavily relies on large-scale data. Thus, deep learning is prone to overfit when only few labeled samples are available. Few-shot learning (FSL) aims to address this issue by learning to generalize to new tasks with few training samples (e.g., one or five samples per class). FSL considers learning good representations crucial to classify accurately using discriminative features. In this study, we propose a framework for few-shot TSC that encodes a time series as different types of images (i.e., Recurrence plot, Markov transition field, and Gramian angular summation/difference field) and train these images to the model using the FSL procedure. Different features of each image enable the model to learn rich information. In addition, we propose temporal-context attention (TCA) and meta-feature fusion (MFF) to maximize the representation ability of these images. TCA incorporates global context of the feature map and highlights pixels having informative relevance with other pixels. After extracting features, MFF refines each feature using different kernels generated based on cross-modality features and fuses the refined features. Finally, the test samples are classified to the nearest class prototype in the embedding space. All experiments are conducted on various N-way K-shot problems. Our framework outperforms state-of-the-art models on 28 standard datasets in the UCR (University of California, Riverside) archive, which is a widely used benchmark dataset in time series classification, from 0.34% up to 29.4%.