Facial Expression Recognition Using Kinect Depth Sensor and Convolutional Neural Networks

Ijjina, Earnest Paul; Mohan, C. Krishna

doi:10.1109/icmla.2014.70

Cited by 32 publications

(11 citation statements)

References 13 publications

(13 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, the patient logs into the system through a face recognition technology embedded in the exercise game using Kinect v2. We used face recognition to identify the user because of the following two reasons: first, it allows natural interaction with the system, with high recognition accuracy [ 34 - 37 ], and second, in the future, we can extend the exercise game system with emotion recognition using the camera [ 38 - 40 ].…”

Section: Methodsmentioning

confidence: 99%

Usability Test of Exercise Games Designed for Rehabilitation of Elderly Patients After Hip Replacement Surgery: Pilot Study

Ling¹,

Meer²,

Yumak³

et al. 2017

JMIR Serious Games

View full text Add to dashboard Cite

BackgroundPatients who receive rehabilitation after hip replacement surgery are shown to have increased muscle strength and better functional performance. However, traditional physiotherapy is often tedious and leads to poor adherence. Exercise games, provide ways for increasing the engagement of elderly patients and increase the uptake of rehabilitation exercises.ObjectiveThe objective of this study was to evaluate Fietsgame (Dutch for cycling game), which translates existing rehabilitation exercises into fun exercise games. The system connects exercise games with a patient’s personal record and a therapist interface by an Internet of Things server. Thus, both the patient and physiotherapist can monitor the patient’s medical status.MethodsThis paper describes a pilot study that evaluates the usability of the Fietsgame. The study was conducted in a rehabilitation center with 9 participants, including 2 physiotherapists and 7 patients. The patients were asked to play 6 exercise games, each lasting about 5 min, under the guidance of a physiotherapist. The mean age of the patients was 74.57 years (standard deviation [SD] 8.28); all the patients were in the recovery process after hip surgery. Surveys were developed to quantitatively measure the usability factors, including presence, enjoyment, pain, exertion, and technology acceptance. Comments on advantages and suggested improvements of our game system provided by the physiotherapists and patients were summarized and their implications were discussed.ResultsThe results showed that after successfully playing the games, 75% to 100% of the patients experienced high levels of enjoyment in all the games except the squats game. Patients reported the highest level of exertion in squats when compared with other exercise games. Lunges resulted in the highest dropout rate (43%) due to interference with the Kinect v2 from support chairs. All the patients (100%) found the game system useful and easy to use, felt that it would be a useful tool in their further rehabilitation, and expressed that they would like to use the game in the future. The therapists indicated that the exercise games highly meet the criteria of motor rehabilitation, and they intend to continue using the game as part of their rehabilitation treatment of patients. Comments from the patients and physiotherapists suggest that real-time corrective feedback when patients perform the exercises wrongly and a more personalized user interface with options for increasing or decreasing cognitive load are needed.ConclusionsThe results suggest that Fietsgame can be used as an alternative tool to traditional motor rehabilitation for patients with hip surgery. Lunges and squats are found to be more beneficial for patients who have relatively better balance skills. A follow-up randomized controlled study will be conducted to test the effectiveness of the Fietsgame to investigate how motivating it is over a longer period of time.

show abstract

Section: Methodsmentioning

confidence: 99%

Usability Test of Exercise Games Designed for Rehabilitation of Elderly Patients After Hip Replacement Surgery: Pilot Study

Ling¹,

Meer²,

Yumak³

et al. 2017

JMIR Serious Games

View full text Add to dashboard Cite

show abstract

“…Some approaches applied 3D scanning [13] and thermal imaging [14] to recording the facial information. Though 3D scanning is accurate and invariant to illumination changes compared to other approaches, it requires specialized expensive equipment and capture in controlled environments [15]. Therefore, Microsoft Kinect as a 3D sensor is an attractive alternative due to its low cost, portability, and applicability in many interactive applications such as games and action recognition.…”

Section: Introductionmentioning

confidence: 99%

“…In [15], depth information was used to recognize facial expressions with open mouth, occlusion of mouth by hand and occlusion by paper. The Gradient direction information of depth data was used as facial features and sent into the convolutional neural network for classification.…”

Section: Introductionmentioning

confidence: 99%

CNN-Based Facial Expression Recognition from Annotated RGB-D Images for Human–Robot Interaction

et al. 2019

Int. J. Human. Robot.

View full text Add to dashboard Cite

Facial expression recognition has been widely used in human computer interaction (HCI) systems. Over the years, researchers have proposed different feature descriptors, implemented different classification methods, and carried out a number of experiments on various datasets for automatic facial expression recognition. However, most of them used 2D static images or 2D video sequences for the recognition task. The main limitations of 2D-based analysis are problems associated with variations in pose and illumination, which reduce the recognition accuracy. Therefore, an alternative way is to incorporate depth information acquired by 3D sensor, because it is invariant in both pose and illumination. In this paper, we present a two-stream convolutional neural network (CNN)-based facial expression recognition system and test it on our own RGB-D facial expression dataset collected by Microsoft Kinect for XBOX in unspontaneous scenarios since Kinect is an inexpensive and portable device to capture both RGB and depth information. Our fully annotated dataset includes seven expressions (i.e., neutral, sadness, disgust, fear, happiness, anger, and surprise) for 15 subjects (9 males and 6 females) aged from 20 to 25. The two individual CNNs are identical in architecture but do not share parameters. To combine the detection results produced by these two CNNs, we propose the late fusion approach. The experimental results demonstrate that the proposed two-stream network using RGB-D images is superior to that of using only RGB images or depth images.

show abstract

“…, SIFT, HOG, LBP) at the frame-by-frame level and train off-the-shelf classifiers for the recognition of AUs at the frame level. Representative approaches include neural networks [33], Bayesian networks [35], support vector machine with single margin [5], [7], [24] or multiple margins [42], boosting based approaches [1], and more recently the end-to-end convolutional neural networks [14], [45]. Dynamic approaches consider temporal information by recognizing AUs at the segment level ( i.e.…”

Section: Introductionmentioning

confidence: 99%

“…CNN has become one of the most powerful machine learning methods in large-scale object detection, image classification [21], [30], and more recently AU detection [14], [17]. Other approaches to AU detection first engineer hand-crafted features and then independently train classifiers.…”

Section: Introductionmentioning

confidence: 99%

Automatic action unit detection in infants using convolutional neural network

Hammal

Chu

Cohn

et al. 2017

2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)

View full text Add to dashboard Cite

Action unit detection in infants relative to adults presents unique challenges. Jaw contour is less distinct, facial texture is reduced, and rapid and unusual facial movements are common. To detect facial action units in spontaneous behavior of infants, we propose a multi-label Convolutional Neural Network (CNN). Eighty-six infants were recorded during tasks intended to elicit enjoyment and frustration. Using an extension of FACS for infants (Baby FACS), over 230,000 frames were manually coded for ground truth. To control for chance agreement, inter-observer agreement between Baby-FACS coders was quantified using free-margin kappa. Kappa coefficients ranged from 0.79 to 0.93, which represents high agreement. The multi-label CNN achieved comparable agreement with manual coding. Kappa ranged from 0.69 to 0.93. Importantly, the CNN-based AU detection revealed the same change in findings with respect to infant expressiveness between tasks. While further research is needed, these findings suggest that automatic AU detection in infants is a viable alternative to manual coding of infant facial expression.

show abstract

Facial Expression Recognition Using Kinect Depth Sensor and Convolutional Neural Networks

Cited by 32 publications

References 13 publications

Usability Test of Exercise Games Designed for Rehabilitation of Elderly Patients After Hip Replacement Surgery: Pilot Study

Usability Test of Exercise Games Designed for Rehabilitation of Elderly Patients After Hip Replacement Surgery: Pilot Study

CNN-Based Facial Expression Recognition from Annotated RGB-D Images for Human–Robot Interaction

Automatic action unit detection in infants using convolutional neural network

Contact Info

Product

Resources

About