A Review of Emotion Recognition Methods From Keystroke, Mouse, and Touchscreen Dynamics

Abstract

Emotion can be defined as a subject's organismic response to an external or internal stimulus event. The responses could be reflected in pattern changes of the subject's facial expression, gesture, gait, eye-movement, physiological signals, speech and voice, keystroke, and mouse dynamics, etc. This suggests that on the one hand emotions can be measured/recognized from the responses, and on the other hand they can be facilitated/regulated by external stimulus events, situation changes or internal motivation changes. It is well-known that emotion has a close relationship with both physical and mental health, usually affecting an individual's and a team's work performance, thus emotion recognition is an important prerequisite for emotion regulation towards better emotional states and work performance. The primary problem in emotion recognition is how to recognize a subject's emotional states easily and accurately. Currently, there are a body of good research on emotion recognition from facial expression, gesture, gait, eye-tracking, and other physiological signals such as speech and voice, but they are all intrusive and obtrusive to some extent. In contrast, keystroke, mouse and touchscreen (KMT) dynamics data can be collected non-intrusively and unobtrusively as secondary data responding to primary physical actions, thus, this paper aims to review the state-of-the-art research on emotion recognition from KMT dynamics and to identify key research challenges, opportunities and a future research roadmap for referencing. In addition, this paper answers the following six research questions (RQs): (1) what are the commonly used emotion elicitation methods and databases for emotion recognition? (2) which emotions could be recognized from KMT dynamics? (3) what key features are most appropriate for recognizing different specific emotions? (4) which classification methods are most effective for specific emotions? ( 5) what are the application trends of emotion recognition from KMT dynamics? (6) which application contexts are of greatest concern?

A Systematic Literature Review on Latest Keystroke Dynamics Based Models

“…In the same year, another study [48] reviewed some existing classification methods, features, and input texts and provided a limitation of the solutions. In recent years, a study [49] surveyed KD-based emotion recognition models. Their effort was to answer the important six research questions for further development of an emotion recognition system using KD attributes.…”

Wireless Sensing Technology Combined with Facial Expression to Realize Multimodal Emotion Recognition

“…In recent years, considering the great influence of emotions on people’s physical and mental health, the work on emotion recognition has a broad application prospect, which is one of the reasons why this field has received close attention from the industry [ 17 , 18 , 19 , 20 , 21 ]. Since emotional fluctuations cause changes in the external expressive reflections of the human body, researchers have extracted emotion-related features through facial expressions [ 22 ], body movements [ 23 , 24 ], and voice intonation [ 25 ].…”

Piezoelectric Touch Sensing and Random-Forest-Based Technique for Emotion Recognition

“…In recent years, attentions start to be paid to using behavioral biometric measures in emotion detection. Compared to physiologic characters, behaviors can be measured using existing peripherals, such as a mouse, a keyboard, or a touch screen [31]. Moreover, behaviors such as typing or swiping can be monitored voicelessly and continuously, as opposed to facial expression or voice, which don't add burdens on users.…”