Facial Position and Expression-Based Human–Computer Interface for Persons With Tetraplegia

Abstract

A human-computer interface (namely Facial position and expression Mouse system, FM) for the persons with tetraplegia based on a monocular infrared depth camera is presented in this paper. The nose position along with the mouth status (close/open) is detected by the proposed algorithm to control and navigate the cursor as computer user input. The algorithm is based on an improved Randomized Decision Tree, which is capable of detecting the facial information efficiently and accurately. A more comfortable user experience is achieved by mapping the nose motion to the cursor motion via a nonlinear function. The infrared depth camera enables the system to be independent of illumination and color changes both from the background and on human face, which is a critical advantage over RGB camera-based options. Extensive experimental results show that the proposed system outperforms existing assistive technologies in terms of quantitative and qualitative assessments.

Touchless typing with head movements captured in thermal spectrum

“…1 Comparison of face images taken at the same moment in unfavourable lighting conditions using visible (left) and thermal (right) camera GUI using head movements. In [5,23] and [37] the position of user's nostrils related to the face region is used. Interestingly, in [5], a depth imaging technique is adapted.…”

“…In [5,23] and [37] the position of user's nostrils related to the face region is used. Interestingly, in [5], a depth imaging technique is adapted. From a depth image, the nose position and the mouth status are detected and used for steering.…”

“…clicking events), different non-rigid motions are employed. Some of the examples include: eye blinks [24,30,37], mouth shape changes (opening, closing or stretching) [5,14,33,36], brows movements [14] or cheeks twitch.…”

Predicting Mouse Cursor Target using Backpropagation Neural Network

“…Kocejko et al [8] and Lupu et al [9] have controlled the mouse cursor by tracking the eye gaze movements of the user. Betke et al [10], Epstein et al [11], Nabati et al [12], Chareonsuk et al [13], Varona et al [14], Bian et al [15], Gorodnichy et al [16], Gyawal et al [17] and Morris et al [18] have avoided the overhead of using and head-mounted devices and high-cost hardware system by capturing the user's head motions through web camera to control the mouse pointer. Fathi et al [19] achieved this by tracking the eye movement whereas Sugano et al [20], Sambrekar et al [21] and M. Nasor et al [22] have attempted tracking the eye gazes.…”

“…Fathi et al [19] achieved this by tracking the eye movement whereas Sugano et al [20], Sambrekar et al [21] and M. Nasor et al [22] have attempted tracking the eye gazes. Betke et al [10], Nabati et al [12], Chareonsuk et al [13], Varona et al [14], Bian et al [15], Gorodnichy et al [16], Fathi et al [19], Hegde et al [23] and Arai et al [24] have developed the camera-based mouse replacement solutions for implementing mouse click events such as single and doubleclicking and dragging. Tracking and converting accurately the facial expression of the user to the mouse operation is still acknowledged as a research challenge and opportunity.…”