Temporal Convolutional Networks for Multiperson Activity Recognition Using a 2-D LIDAR

Luo, Fei; Poslad, Stefan; Bodanese, Eliane

doi:10.1109/jiot.2020.2984544

Cited by 53 publications

(31 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Concerning ML methods used for activity recognition, CNN's appear to be a popular choice [56]- [58]. Regrettably, point cloud data is not a natural input to CNN.…”

Section: Related Workmentioning

confidence: 99%

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Roche

De-Silva

Hook

et al. 2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Increasingly, the task of detecting and recognising the actions of a human has been delegated to some form of neural network processing camera or wearable sensor data. Due to the degree to which the camera can be affected by lighting and wearable sensors scantiness, neither one modality can capture the required data to perform the task confidently. That being the case, range sensors like Light Detection and Ranging (LiDAR) can complement the process to perceive the environment more robustly. Most recently, researchers have been exploring ways to apply Convolutional Neural Networks to 3-Dimensional data. These methods typically rely on a single modality and cannot draw on information from complementing sensor streams to improve accuracy. This paper proposes a framework to tackle human activity recognition by leveraging the benefits of sensor fusion and multimodal machine learning. Given both RGB and point cloud data, our method describes the activities being performed by subjects using regions with a convolutional neural network (R-CNN) and a 3-Dimensional modified Fisher vector network. Evaluated on a custom captured multimodal dataset demonstrates that the model outputs remarkably accurate human activity classification (90%). Furthermore, this framework can be used for sports analytics, understanding social behaviour, surveillance, and perhaps most notably by Autonomous Vehicles (AV) to datadriven decision-making policies in urban areas and indoor environments.

show abstract

“…Concerning ML methods used for activity recognition, CNN's appear to be a popular choice [56]- [58]. Regrettably, point cloud data is not a natural input to CNN.…”

Section: Related Workmentioning

confidence: 99%

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Roche

De-Silva

Hook

et al. 2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

show abstract

“…Recently, researchers proposed approaches that replace the layers based on RNNs with layers based on Temporal Convolutional Networks (TCNs) for several types of sequence modelling problems [ 14 , 15 , 16 , 17 ]. The concept of TCN was first introduced by Lea et al [ 10 ].…”

Section: Related Workmentioning

confidence: 99%

Comparison between Recurrent Networks and Temporal Convolutional Networks Approaches for Skeleton-Based Action Recognition

Nan

Trăşcău

Florea

et al. 2021

Sensors

View full text Add to dashboard Cite

Action recognition plays an important role in various applications such as video monitoring, automatic video indexing, crowd analysis, human-machine interaction, smart homes and personal assistive robotics. In this paper, we propose improvements to some methods for human action recognition from videos that work with data represented in the form of skeleton poses. These methods are based on the most widely used techniques for this problem—Graph Convolutional Networks (GCNs), Temporal Convolutional Networks (TCNs) and Recurrent Neural Networks (RNNs). Initially, the paper explores and compares different ways to extract the most relevant spatial and temporal characteristics for a sequence of frames describing an action. Based on this comparative analysis, we show how a TCN type unit can be extended to work even on the characteristics extracted from the spatial domain. To validate our approach, we test it against a benchmark often used for human action recognition problems and we show that our solution obtains comparable results to the state-of-the-art, but with a significant increase in the inference speed.

show abstract

“…However, one work by Luo et al indicates that classical 2D-TCNs can perform equally well and even outperform approaches based on LSTM cells and HMM for gesture recognition tasks. [42] This paper presents a combination of TCN and CNN models to improve energy efficiency, reduce memory requirements and maximize the accuracy of gesture recognition using sensor data from a short-range RADAR. The hardware implementation and the benefits of the combination of TCN have briefly been discussed in the authors' previous work.…”

Section: A Image-based Gesture Recognitionmentioning

confidence: 99%

TinyRadarNN: Combining Spatial and Temporal Convolutional Neural Networks for Embedded Gesture Recognition with Short Range Radars

Scherer,

Magno,

Erb

et al. 2020

Preprint

View full text Add to dashboard Cite

This work proposes a low-power high-accuracy embedded hand-gesture recognition algorithm targeting batteryoperated wearable devices using low power short-range RADAR sensors. A 2D Convolutional Neural Network (CNN) using range frequency Doppler features is combined with a Temporal Convolutional Neural Network (TCN) for time sequence prediction.The final algorithm has a model size of only 46 thousand parameters, yielding a memory footprint of only 92 KB. Two datasets containing 11 challenging hand gestures performed by 26 different people have been recorded containing a total of 20,210 gesture instances. On the 11 hand gesture dataset, accuracies of 86.6% (26 users) and 92.4% (single user) have been achieved, which are comparable to the state-of-the-art, which achieves 87% (10 users) and 94% (single user), while using a TCN-based network that is 7500x smaller than the state-of-the-art. Furthermore, the gesture recognition classifier has been implemented on Parallel Ultra-Low Power Processor, demonstrating that real-time prediction is feasible with only 21 mW of power consumption for the full TCN sequence prediction network.

show abstract

Temporal Convolutional Networks for Multiperson Activity Recognition Using a 2-D LIDAR

Cited by 53 publications

References 42 publications

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Comparison between Recurrent Networks and Temporal Convolutional Networks Approaches for Skeleton-Based Action Recognition

TinyRadarNN: Combining Spatial and Temporal Convolutional Neural Networks for Embedded Gesture Recognition with Short Range Radars

Contact Info

Product

Resources

About