A machine learning approach for gesture recognition with a lensless smart sensor system

Norman, Niccolo; Urru, Andrea; Abraham, Lizy; Walsh, Michael; Tedesco, Salvatore; Cenedese, Angelo; Susto, Gian Antonio; O’Flynn, Brendan

doi:10.1109/bsn.2018.8329677

Cited by 12 publications

(8 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Optoelectronics have been widely and consistently used in robotics over the recent years, particularly in the research field of collaborative systems and are shown to increase the safety of human operators. In the future, the price drop of optoelectronic sensors and the release of more compact and easier to implement hybrid and data fusion solutions, as well as next-generation wearable lens-less cameras [ 83 , 84 , 85 ], will lead to fewer obstructions in jobsites and improve the practicality of camera-based approaches in other industry sectors.…”

Section: Discussionmentioning

confidence: 99%

Motion Capture Technology in Industrial Applications: A Systematic Review

Menolotto

Komaris

Tedesco

et al. 2020

Sensors

Self Cite

169

View full text Add to dashboard Cite

The rapid technological advancements of Industry 4.0 have opened up new vectors for novel industrial processes that require advanced sensing solutions for their realization. Motion capture (MoCap) sensors, such as visual cameras and inertial measurement units (IMUs), are frequently adopted in industrial settings to support solutions in robotics, additive manufacturing, teleworking and human safety. This review synthesizes and evaluates studies investigating the use of MoCap technologies in industry-related research. A search was performed in the Embase, Scopus, Web of Science and Google Scholar. Only studies in English, from 2015 onwards, on primary and secondary industrial applications were considered. The quality of the articles was appraised with the AXIS tool. Studies were categorized based on type of used sensors, beneficiary industry sector, and type of application. Study characteristics, key methods and findings were also summarized. In total, 1682 records were identified, and 59 were included in this review. Twenty-one and 38 studies were assessed as being prone to medium and low risks of bias, respectively. Camera-based sensors and IMUs were used in 40% and 70% of the studies, respectively. Construction (30.5%), robotics (15.3%) and automotive (10.2%) were the most researched industry sectors, whilst health and safety (64.4%) and the improvement of industrial processes or products (17%) were the most targeted applications. Inertial sensors were the first choice for industrial MoCap applications. Camera-based MoCap systems performed better in robotic applications, but camera obstructions caused by workers and machinery was the most challenging issue. Advancements in machine learning algorithms have been shown to increase the capabilities of MoCap systems in applications such as activity and fatigue detection as well as tool condition monitoring and object recognition.

show abstract

Section: Discussionmentioning

confidence: 99%

Motion Capture Technology in Industrial Applications: A Systematic Review

Menolotto

Komaris

Tedesco

et al. 2020

Sensors

Self Cite

169

View full text Add to dashboard Cite

show abstract

“…It validated the performance of the Rambus LSS system in gesture recognition applications. A more detailed description of different machine learning approaches and an analysis of the results are provided in [ 51 ]. In this work, the classification accuracy as a function of LED position using Random Forest training [ 52 ] is evaluated to validate the performance.…”

Section: Resultsmentioning

confidence: 99%

“…The feature extraction consists of a procedure to gather the principal characteristics of a gesture to represent it as a unique vector that is able to describe it. A frame-based descriptor approach is used for the same [ 50 , 51 ]. As illustrated in [ 50 ], the frame-based descriptor is well suited for extracting features from inertial sensors.…”

Section: Methodsmentioning

confidence: 99%

“…Even if the system is capable of a frame rate equal to 40 fps, it is sufficient to represent a gesture by down sampling to 20 fps. The mathematical modeling of a gesture descriptor is well described in the latest work of authors [ 51 ]. The dataset is then properly formulated and applied in the training and testing phases of a Random Forest (RF) model [ 52 ] to validate the performance of gesture recognition.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Hand Tracking and Gesture Recognition Using Lensless Smart Sensors

Abraham

Urru

Normani

et al. 2018

Sensors

Self Cite

View full text Add to dashboard Cite

The Lensless Smart Sensor (LSS) developed by Rambus, Inc. is a low-power, low-cost visual sensing technology that captures information-rich optical data in a tiny form factor using a novel approach to optical sensing. The spiral gratings of LSS diffractive grating, coupled with sophisticated computational algorithms, allow point tracking down to millimeter-level accuracy. This work is focused on developing novel algorithms for the detection of multiple points and thereby enabling hand tracking and gesture recognition using the LSS. The algorithms are formulated based on geometrical and mathematical constraints around the placement of infrared light-emitting diodes (LEDs) on the hand. The developed techniques dynamically adapt the recognition and orientation of the hand and associated gestures. A detailed accuracy analysis for both hand tracking and gesture classification as a function of LED positions is conducted to validate the performance of the system. Our results indicate that the technology is a promising approach, as the current state-of-the-art focuses on human motion tracking that requires highly complex and expensive systems. A wearable, low-power, low-cost system could make a significant impact in this field, as it does not require complex hardware or additional sensors on the tracked segments.

show abstract

“…The algorithms for HAR can be classified into shallow and deep learning methods. Common shallow methods in HAR include SVM [13], [20], [23], k-nearest neighbors (kNN) [16], [24], linear discriminant analysis (LDA) [9], and random forest (RF) [21]. Deep learning approaches, such as LSTM [7], [15], CNN-LSTM [25], [27], CNN [22], and convLSTM [26], have shown impressive leaps in performance compared to their shallow counterparts by learning to automatically extract features from raw sensor data, thus dropping the need for having human experts to provide hand-engineered features.…”

Section: Background a Human Activity Recognition (Har)mentioning

confidence: 99%

Improving Cross-Subject Activity Recognition via Adversarial Learning

Leite

Xiao

2020

IEEE Access

View full text Add to dashboard Cite

Deep learning has been widely used for implementing human activity recognition from wearable sensors like inertial measurement units. The performance of deep activity recognition is heavily affected by the amount and variability of the labeled data available for training the deep learning models. On the other hand, it is costly and time-consuming to collect and label data. Given limited training data, it is hard to maintain high performance across a wide range of subjects, due to the differences in the underlying data distribution of the training and the testing sets. In this work, we develop a novel solution that applies adversarial learning to improve cross-subject performance by generating training data that mimic artificial subjects-i.e. through data augmentation-and enforcing the activity classifier to ignore subject-dependent information. Contrary to domain adaptation methods, our solution does not utilize any data from subjects of the test set (or target domain). Furthermore, our solution is versatile as it can be utilized together with any deep neural network as the classifier. Considering the open dataset PAMAP2, nearly 10% higher crosssubject performance in terms of F1-score can be achieved when training a CNN-LSTM-based classifier with our solution. A performance gain of 5% is also observed when our solution is applied to a stateof-the-art HAR classifier composed of a combination of inception neural network and recurrent neural network. We also investigate different influencing factors of classification performance (i.e. selection of sensor modalities, sampling rates and the number of subjects in the training data), and summarize a practical guideline for implementing deep learning solutions for sensor-based human activity recognition.

show abstract

A machine learning approach for gesture recognition with a lensless smart sensor system

Cited by 12 publications

References 5 publications

Motion Capture Technology in Industrial Applications: A Systematic Review

Motion Capture Technology in Industrial Applications: A Systematic Review

Hand Tracking and Gesture Recognition Using Lensless Smart Sensors

Improving Cross-Subject Activity Recognition via Adversarial Learning

Contact Info

Product

Resources

About