Gesture Recognition in RGB Videos Using Human Body Keypoints and Dynamic Time Warping

Abstract

Gesture recognition opens up new ways for humans to intuitively interact with machines. Especially for service robots, gestures can be a valuable addition to the means of communication to, for example, draw the robot's attention to someone or something. Extracting a gesture from video data and classifying it is a challenging task and a variety of approaches have been proposed throughout the years. This paper presents a method for gesture recognition in RGB videos using OpenPose to extract the pose of a person and Dynamic Time Warping (DTW) in conjunction with One-Nearest-Neighbor (1NN) for time-series classification. The main features of this approach are the independence of any specific hardware and high flexibility, because new gestures can be added to the classifier by adding only a few examples of it. We utilize the robustness of the Deep Learning-based OpenPose framework while avoiding the data-intensive task of training a neural network ourselves. We demonstrate the classification performance of our method using a public dataset.

Student Behavior Recognition System for the Classroom Environment Based on Skeleton Pose Estimation and Person Detection

“…To classify human activities, the aforementioned authors developed an LSTM-RNN model by combining the long short-term memory (LSTM) model with the recurrent neural network (RNN) model. Schneider et al [ 24 ] presented a method for recognizing human gestures. In this method, the OpenPose framework is used to detect human poses, and the one-nearest-neighbor classifier is used to classify the detected poses.…”

“…Many difficulties arise in the broad implementation of the aforementioned method due to the replacement of the input camera in the classroom. To resolve this issue, the OpenPose framework [ 21 ] is used to estimate two-dimensional (2D) human poses [ 22 , 23 , 24 ]. However, the limitation of this approach is that incorrect connections occur in pose estimation in highly crowded areas such as classrooms.…”

Uma Abordagem Baseada No Aprendizado De Máquina Interativo E Interação Natural Para Apoio À Reabilitação Física

“…Para tanto, podem ser usados modelos de aprendizado de máquina, como Hidden Markov Model (HMM) 10 , Recurrent Neural Network (RNN) 11 , Convolutional Neural Network (CNN) 12 , entre outros; ou algoritmos de alinhamento temporal, como o algoritmo DTW [Berndt and Clifford 1994], distância Euclidiana (ED) 13 ou suas variantes. Nesta tese foram usados o algoritmo DTW em conjunto com o algoritmo k-nearest neighbors (kNN) para a classificação da série temporal, como nos trabalhos de [Schneider et al 2019, Ribó et al 2016, Chandrasekhar and Mhala 2018, Chen et al 2020.…”

“…Posteriormente, realizamos duas etapas de processamento para permitir a invariância do tamanho de cada pessoa e reduzir a dimensionalidade dos dados. Primeiro, para permitir a invariância do tamanho e posição do paciente, normalizamos as coordenadas do ponto como apresentado nos trabalhos de [Celebi et al 2013] e [Schneider et al 2019]. Aqui, reposicionamos o ponto-chave do pescoço na origem e transladamos todos os pontos circundantes em correspondência.…”

“…Em contrapartida, os demais carecem de informações úteis para o exercício correspondente. Portanto, nesta etapa selecionamos os IDs dos pontos cuja variação no tempo é superior a um limite e, descartamos os pontos não informativos [Schneider et al 2019].…”