KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion Recognition

Abstract

In this paper, we propose an integrated approach to robot vision: a key frame-based skeleton feature estimation and action recognition network (KFSENet) that incorporates action recognition with face and emotion recognition to enable social robots to engage in more personal interactions. Instead of extracting the human skeleton features from the entire video, we propose a key frame-based approach for their extraction using pose estimation models. We select the key frames using the gradient of a proposed total motion metric that is computed using dense optical flow. We use the extracted human skeleton features from the selected key frames to train a deep neural network (i.e., the double-feature double-motion network (DDNet)) for action recognition. The proposed KFSENet utilizes a simpler model to learn and differentiate between the different action classes, is computationally simpler and yields better action recognition performance when compared with existing methods. The use of key frames allows the proposed method to eliminate unnecessary and redundant information, which improves its classification accuracy and decreases its computational cost. The proposed method is tested on both publicly available standard benchmark datasets and self-collected datasets. The performance of the proposed method is compared to existing state-of-the-art methods. Our results indicate that the proposed method yields better performance compared with existing methods. Moreover, our proposed framework integrates face and emotion recognition to enable social robots to engage in more personal interaction with humans.

Analyzing audiovisual data for understanding user's emotion in human−computer interaction environment

“…It is concluded that key-frame extraction is highly significant in processing video data. In general, existing key-frame extraction methods consist of shot boundary detection (Fei et al , 2017; Mehmood et al , 2016), frame image clustering (Wu et al , 2017; Gharbi et al , 2017), motion analysis (Le et al , 2022; Anderson and McOwan, 2006) and visual content analysis (Panagiotakis et al , 2009).…”

“…Key-frame extraction aims to extract a set of images from original video, which are expected to be an approximate representation of the visual contents of the entire video (Huang and Wang, 2019). Traditional key-frame extraction methods consist of shot boundary detection (Fei et al , 2017; Mehmood et al , 2016), frame image clustering (Wu et al , 2017; Gharbi et al , 2017), motion analysis (Le et al , 2022; Anderson and McOwan, 2006) and visual content analysis (Panagiotakis et al , 2009). The shot boundary-based methods are simple and computationally efficient, but they can only select a fixed number of images as key-frames without considering the content complexity (Fei et al , 2017; Mehmood et al , 2016).…”

“…Clustering-based methods are very popular, but they need to preestablish the number of clusters and cannot preserve the sequential information of key-frames (Wu et al , 2017; Gharbi et al , 2017). The motion-based methods, which are mainly designed based on the calculation of the optical flow, not only ignore the cumulative changes in content but also suffer from high computation and weak robustness (Le et al , 2022; Anderson and McOwan, 2006). The visual content-based methods mainly depend on the changes of content (such as texture and color) to extract discriminative frames (Panagiotakis et al , 2009), which is simple but effective in extracting key-frames (Yong et al , 2013).…”

Various frameworks for integrating image and video streams for spatiotemporal information learning employing 2D-3D Residual networks for human action recognition

“…The keyframe is the frame with the highest histogram correlation from a set of consecutive frames. Choosing the number of consecutive frames (shot) affects the keyframe extraction accuracy [74,75], where If the consecutive frames are small sets of frames the variation of frame histograms will be very small to identify one keyframe, and if we use a large set of consecutive frames there could be more than one keyframe in the shot and we only extract one and neglect others. We explore the keyframe extraction method for a variant video clip length: 5 frames per clip and 16 frames per clip.…”

Facial Emotion Recognition Analysis Based on Age-Biased Data

“…However, emotion classification is still a challenging task. Convolutional neural networks (CNNs) perform face normalization, facial expressions, and emotional classification using real images as their main functions and are frequently adopted and used in computer vision applications [3][4][5][6]. The accuracy of the CNN-based emotion classification system has been improved through pre-or post-processing [2,5,7] and the development of new algorithms in the architecture.…”